ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation#6323
ARTEMIS-5573 and ARTEMIS-5975 Improve AMQP Size estimation#6323clebertsuconic wants to merge 2 commits into
Conversation
f70cda5 to
be90ac4
Compare
0611d5a to
03b7176
Compare
94efe35 to
70beb3a
Compare
tabish121
left a comment
There was a problem hiding this comment.
Overall looks good, nice simplification of some of the message handling
09bb86b to
540ad20
Compare
|
The reason I had to add a V4 is because of Paging: On this part here, I encode the number of queues and the queueIDs: At the point of the encoder I don't have any reference to the number of bytes used by its own decoder. I'm adding one on V4 now, if in the future anything else is added, we will stop reading at that marker. |
1cb4403 to
fa1128e
Compare
|
I intend to squash the second commit on the first as soon some review is done on the new persister. |
|
I'm setting it as ready to review. but this commit should be squashed before merged. |
|
I needed to add versioning to AMQPLargeMessagePersister as well. I could choose to skip memory calculations on large message... however I will prefer to keep it the same way as StandardMessage. this is also better for future additions |
|
I added a commit addressing the comments, let me know when I can squash it please |
|
I added another commit to be squashed after review |
|
when I worked on this I wasn't planning to rewrite persistence for AMQP messages. I will work on something for those persisters. |
|
I am now using a MacCodec instead of a streamed bytes. I am not using a Hashmap to store records in between, but there's a codec that will store it like it was a hashMap. I also added a clustering compatibility tests since the persistence is used with clustering. |
gemmellr
left a comment
There was a problem hiding this comment.
Gave it a skim over time, havent closely looked at most of it, but still had various comments so here they are. Most important one being, I do not think this should be in 2.54.0 at this stage.
| private static final ThreadLocal<AMQPMessageMapCodec> threadLocal = ThreadLocal.withInitial(AMQPMessageMapCodec::new); | ||
|
|
||
| public static AMQPMessageMapCodec getInstance() { | ||
| AMQPMessageMapCodec reader = threadLocal.get(); | ||
| return reader; | ||
| } |
There was a problem hiding this comment.
Possibly over-thinking it, but the this got me wondering if rather than a subclass, this would work nicer as a method that took a processor / callback-handler style argument. The persister would then just need 1 of those while it worked during startup and then it would go away when done, unlike this. You could also then separately test the codec class (current parent) and the processor/callback fully that way rather than e.g creating a test subclass to test the parent bits, and not really directly testing this class at all [it seems so far].
The large and regular persister could still use the same processor and then just toggle a flag to indicate which behaviour to give.
There was a problem hiding this comment.
@gemmellr I can do that. although I'm using a thread local now because of the state I needed to store before creating a message.
There's one changed I wanted to make on the constructors for AMQPStandardMessage and AMQPLargeMessage, that I could keep the implementation stateless, and just post data directly on the message. then I wouldn't need a ThreadLocal and I could just work with the message directly.
There was a problem hiding this comment.
not sure about a callback. I tried that before I sent the PR and I didn't like it.
There was a problem hiding this comment.
@gemmellr I played with extension versus plugin.. it doesn't change much in the code.
We will still have a single ThreadLocal... I will keep it as extension for now
There was a problem hiding this comment.
I dont recall any need for a ThreadLocal in the approach my comment wondered about, pretty sure removing that that was what made me think of it. Unclear what would you be storing in it with the state in the processor/callback and the rest being static methods or maybe a singleton. It also seemed like it would fairly significantly change to the code and the [direct] testing, particularly of the current subclass.
There was a problem hiding this comment.
The main issue is that I need to know the messageFormat before creating an instance of AMQPStandardMessage or AMQPLargeMessage.
I have relaxed this a bit now and won't require any threadLocals any longer and will just parse it directly onto the mesasge object. (I will be push the commit shortly)
There was a problem hiding this comment.
@gemmellr I tried decoding directly into the message but it made things worse.
Historically we've always created the message first, then applied reload. Inverting this for AMQPStandardMessage seemed doable but required a lot of new code. For AMQPLargeMessage it's not feasible at all—the messageID is tied to a file, and adding reload methods just creates more problems than it solves.
I'd keep the current approach with ThreadLocal state. If you want the codec stateless, we'd still need a ThreadLocal or temporary instance to hold the data before message instantiation anyway.
|
@gemmellr I am marking this as draft because of the versioning.. whenever we decide to merge it I will change it and make it ready to commit |
- making it immutable to avoid races after updates like we had in the past - avoiding scanning after reloading by persisting extra data on storage
| private static final ThreadLocal<AMQPMessageMapCodec> threadLocal = ThreadLocal.withInitial(AMQPMessageMapCodec::new); | ||
|
|
||
| public static AMQPMessageMapCodec getInstance() { | ||
| AMQPMessageMapCodec reader = threadLocal.get(); | ||
| return reader; | ||
| } |
There was a problem hiding this comment.
I dont recall any need for a ThreadLocal in the approach my comment wondered about, pretty sure removing that that was what made me think of it. Unclear what would you be storing in it with the state in the processor/callback and the rest being static methods or maybe a singleton. It also seemed like it would fairly significantly change to the code and the [direct] testing, particularly of the current subclass.
|
I have added a commit addressing some of the comments.. I still need to finish this , but it will be tomorrow. for now I have made this a draft again. |
|
Only point I have open now is the discussion on the ThreadLocal for AMQPMessageMetadataPersister. Making it stateless will require intermediate state somewhere else... or a bigger change in the order messages are created versus reloaded. (I want to avoid such inversion at this point) |
No description provided.